Knowledge Discovery 25.4.0
25.4 Release Notes

# NiFi Ingest

## 25.4.0

### New Features

- The universal redaction example dataflow has been improved. The flow definition now enables partitioning so that long documents are partitioned into pages rather than all of the content being combined into one very tall image. The template also outputs the redacted content as a PDF file rather than an image, because the PDF format supports multiple pages. The PDF also contains the text (behind the page image) such that the text can be highlighted and copied, or searched from within a PDF viewer application.
- The AnalyzeMedia processor has a guided setup wizard that can help you to build a Media Server session configuration for several common use-cases.
- The Eduction processor advanced UI has a new tab named "Grammar Packs", which allows you to upload zip files containing Eduction grammars and related files such as post-processing Lua scripts and documentation.
- The KeyViewRouteOnFileType guided setup wizard can help you select preset groups of file formats, for example "Media (OCR)" for all file formats that can be processed by optical character recognition or "Media (STT)" for all file formats that can be processed by speech-to-text.
- The ExecuteDocumentPython processor has been improved. Python modules can be installed to a persistent folder. The built-in documentation for the processor has been improved, and new sample scripts have been added to the advanced UI. It is now possible to modify documents without using callback functions.

### Resolved Issues

There were no resolved issues in this release.

### Notes

- The following NiFi Ingest media processors, which were deprecated in version 25.1, have been removed:

  - FaceDetection
  - ObjectClassRecognition
  - ObjectRecognition
  - OpticalCharacterRecognition
  - PIIImage
  - Redaction
  - SpeechToText
  - VehicleAnalysis

  All of these operations can be performed using the AnalyzeMedia processor.

## 25.3.0

### New Features

- A new RouteDocument processor has been added, so that you can route documents by metadata field values, document properties such as the number of file or filename parts, and custom conditions defined with NiFi expression language. The processor has a guided setup interface that helps you define the conditions for matching documents.
- A new UpdatePersistentCounter processor has been added, so that you can monitor the number of FlowFiles passing a point in the dataflow. This processor is similar to the "UpdateCounter" processor that is supplied with Apache NiFi, except that it stores the counter value on disk. This means that the processor can continue counting even if the NiFi instance is restarted.
- The embedded web browser used in some NiFi Ingest processors (such as ContentFromHTML and RenderHTML) has been upgraded to Chromium 128.
- The RenderHTML processor has additional properties that allow you to configure the viewport height and width.
- The MediaService controller service has a new property named "GPU Device ID", so that you can specify the GPU to use for analysis when the server has more than one GPU.
- The AnalyzeMedia processor has a new property named "Remove document content parts". This specifies whether to remove `contentfile`, `contentfilename`, and `externalfile` parts that were present in incoming documents. Redaction is an example use-case in which you might want to remove this data. You could configure the processor to add a redacted version of an image and remove the original.
- When it encounters a file format that is not supported, the AnalyzeMedia processor adds an error message to the document metadata (earlier versions of the processor generated no output). For each part of a FlowFile that cannot be analyzed, the output XML includes a field `idol_media/analyze_media/error`.
- The embedded version of Python (in the ExecuteDocumentPython processor) has been updated to 3.13.2. Free-threading is supported and can be configured through options set in the `nifi.properties` configuration file.

### Resolved Issues

There were no resolved issues in this release.

### Notes

- The system requirements have changed: running Knowledge Discovery on Linux now requires GLIBC_2.34. You can run Knowledge Discovery on, for example, RHEL 9 or Ubuntu 22.04, but earlier versions are no longer supported. For a full list of supported platforms and operating systems, refer to the Getting Started Guide.
- In the MediaService controller service, the "KeyView Service" property is now mandatory (it must be set).

## 25.2.0

### New Features

- The Document Registry controller service has an advanced UI that you can use to inspect the registry.
- All processors that have a DocumentRegistryService configured will now automatically unregister documents that are transferred to a terminated relationship.
- The StandardizeMetadata processor can display, in the advanced UI, all of the document metadata fields that it has encountered. You can choose to display fields collated from either input or output documents. This can help you configure the processor, by showing you fields that you might want to standardize. To enable this feature, set the new property "Collate Metadata" to "true".
- The Eduction processor advanced UI has a new tab ("TESTER") that you can use for testing and troubleshooting. The UI can extract matches from text or XML metadata that you supply (or from the most recently processed document), and show you the results.
- A new processor, named EnrichFromLookup, makes it easier to add additional data to a document based on an existing field value. This processor provides a way to define lookups that depend on static data contained in a comma- or tab-separated file. An example configuration is included, which adds a latitude, longitude, and place name to a document that only contains a postcode.

### Resolved Issues

- The Eduction processor could generate an invalid Eduction configuration when the guided setup wizard was used and multiple entities were configured to be written to a single document field.
- Attempting to view a document in the output queue of a File System Connector could result in the following error when using a NiFi 2.x instance that was part of a cluster: "HTTP ERROR 500 java.lang.IllegalArgumentException: Unable to determine the which node has the content."

## 25.1.0

### New Features

- NiFi Ingest components are available for Apache NiFi 2.x.

  TIP: The Knowledge Discovery component (.NAR) files for NiFi 1.x are not compatible with NiFi 2.x so you must download and deploy new .NAR files. The Knowledge Discovery .NAR files that support NiFi 2.x include the string `nifi2` in the file name.

- A new controller service, `IdolLoggingServiceImpl`, makes it easier to configure logging for your NiFi Ingest components. You can configure logging without having to edit the `nifi.properties` configuration file.

- FlowFiles that represent Knowledge Discovery documents support a new type of FlowFile part, named `externalfile`. An `externalfile` part contains a reference to a file that is stored by an external storage provider such as Amazon S3, Azure Blob Storage, or Google Cloud Storage.

  - A new controller service, `IdolFlowFileServiceImpl`, manages a connection to an external storage provider.
  - A new processor, `ConvertDocumentFile`, provides a way to convert file content from one type of storage to another. For example, the processor can take embedded binary content or a local file path, upload the file to external storage, and replace the `contentfile` or `contentfilename` part with an `externalfile` part.

  This feature improves support for running NiFi Ingest in the cloud, by allowing you to store FlowFile file content in cloud storage.

- Improved support for running multiple Python scripts concurrently, particularly in container environments when deploying multiple templates that require Python.

  IMPORTANT: The path of the Python interpreter is now configured by a property named `idol.components.python.home`, in the `nifi.properties` file. The default value of this property uses an embedded Python installation that is included with NiFi Ingest, so you should not need to set it. The `IdolPythonServiceImpl` controller service is no longer required and has been deleted. In your NiFi dataflow:

  - The IDOL Python Service can be deleted.
  - The IDOL Python Service property, on any ExecuteDocumentPython or DocumentEmbeddings processor, must be deleted.

### Resolved Issues

There were no resolved issues in this release.

## 24.4.0

### New Features

- A new controller service, `IdolOEMLicenseServiceImpl`, provides a way to license IDOL components using an OEM license key. In an OEM-licensed environment, actions are encrypted using ACI encryption. This service provides decryption for incoming actions (received by the `HandleAciRequest` processor), and enables ACI encryption for outgoing actions.
- The AnalyzeMedia processor can run an XSL transform to take data from the document metadata and write text to the document content. For example, you can take the output from OCR or speech-to-text and use this to populate the document content.
- The DocumentEmbeddings processor now downloads LLM models at start up, rather than when they are first used. This change prevents long request times when there are a large amount of requests sent in a short time frame before the model has been downloaded.

### Resolved Issues

- The RenderHTML processor could generate blank images.
- The DatabaseServiceImpl controller service could generate a validation error ("Sensitive Dynamic Properties [...] configured but not supported") when dynamic properties were used to configure credentials within the database connection string.

### Notes

- The dynamic property `adv:EnableReport` (available on connector processors to enable error reporting) has been removed. The report relationship is now available by default, and is terminated by default.

## 24.3.0

### New Features

- There is a new controller service named `IdolPythonServiceImpl`, which provides the resources that are necessary for IDOL Ingest components to execute Python scripts. The use of a controller service allows multiple IDOL Ingest processors to use the same Python interpreter.
- The IDOL document viewer can display text and image files that are embedded in, or are referenced by, a FlowFile. (You can access the document viewer by listing the contents of a queue and then selecting a FlowFile to view). This feature supports `ContentFile` or `ContentFilename`parts with a MIME type of `text/*` or `image/*`. The viewer displays up to 8 KB of text but you also have the option to download the complete file.

- The following improvements have been made to the Document embeddings processor that you can use to generate embeddings:

  - You can now use a specific revision of a model from Hugging Face, by setting the new Model Revision parameter.

  - You can now use a private model from Hugging Face, by setting the new Access Token parameter.

  - You can now use only an offline (cached) version of your model, rather than downloading the latest, by setting the new Offline Mode parameter.

  - You can now set a limit on the amount of data to generate embeddings for, by setting the new Data Limit parameter.

- The ContentFromHTML processor can process HTML contained in document metadata fields. To do this, set this new property "Source Metadata Fields XPath" to specify the fields to process.

- The AnalyzeMedia processor supports NiFi expression language for the properties "Config" and "Model Files". For example, if you want to run speech-to-text the source language could be specified by a property on the incoming FlowFile.

  To improve security, the processor has also been updated so that by default, it can only write output to a FlowFile. If you want to write output to directories on the NiFi host, you must set the new property "External Output Directories" in your Media Service.

### Resolved Issues

- IDOL Connector "List" processors failed to escape some attributes in FlowFiles, which could result in XML parsing errors.

- The Document embeddings processor could exit unexpectedly if given input consisting only of newlines.

## 24.2.0

### New Features

- The AnalyzeMedia processor that was introduced in IDOL 24.1 now accepts a user-defined session configuration. The configuration is similar to an IDOL Media Server session configuration, with some exceptions. For example, you must ingest data using the `NiFiInput` engine, and output data using the `NiFiOutput` engine. For more information, refer to the IDOL NiFi Ingest documentation.

- The method for using LLMs to generate embeddings in the Document Embeddings service has been improved.

  Rather than create your own model files by using a script, NiFi can now download and cache the models directly from Hugging Face. This change means that NiFi supports a wider range of models, and it does not require the python script or external python libraries.

  To use LLMs, you must now set the Model Name parameter in your Document Embeddings service configuration to specify the model to use from Hugging Face. You can also optionally set Cache Directory to the location that Answer Server must use to store the cached model files.

  For information about the models that you can use for different configurations, refer to the NiFi Ingest Help.

  IMPORTANT: As part of this change, the Model Path and Tokenizer Path parameters have been removed, and are no longer supported.

- In the Document Embeddings service, you can now control the precision of the generated embeddings, by using the Embedding Precision parameter. This parameter sets the number of decimal places to use in the embedding values.

- In the Document Embeddings service, you can now configure embedding models and generative LLMs to use a CUDA-compatible GPU device, by setting the new Device parameter to cuda.

- The advanced configuration UI for the NISTRDSFilter processor includes a cancel button, so that you can cancel the download of NIST RDS hash sets.

### Resolved Issues

There were no resolved issues in this release.

## 24.1.0

### New Features

- The ExecuteDocumentPython processor can install and use third-party modules. There is a new "Packages" tab in the advanced configuration interface that you can use to install modules.
- The Eduction processor has an improved guided setup wizard that makes it easier to select entities.
- A new AnalyzeMedia processor has been added to the IDOL NiFi Media components. This processor can run analysis on rich media files without an IDOL Media Server (all analysis takes place within NiFi). The processor can run one of several pre-set configurations that perform tasks such as Face detection and recognition, OCR, or Speech-To-Text.
- A new DocumentEmbeddings processor has been added to allow you to generate vector embeddings that you can include in your documents to index into IDOL. You can download this processor in the separate IDOL Document Embeddings package.
- The Redaction processor can be used for highlighting. It has a new property named "Opacity", that accepts a value from zero to one. With the default value (1), redacted regions are completely opaque. Values between zero and one produce partially opaque regions, meaning that you can use the processor for highlighting.

### Resolved Issues

There were no resolved issues in this release.

## 23.4.0

### New Features

- A new type of controller service has been introduced to simplify OAuth authentication. The OAuthServiceImpl is a NiFi controller service that provides a suitable endpoint for all IDOL Ingest processors that need to provide an OAuth redirect URL. IDOL Ingest processors can provide their own redirect URLs but these include component names and version numbers, so you might need to update your OAuth applications when you upgrade your IDOL NiFi Ingest components. Using the OAuthServiceImpl, you can set up your OAuth applications with a common redirect URL that does not change.
- A new ExecuteDocumentPython processor has been added, so that you can create and run Python scripts that interact with your FlowFiles and IDOL documents.
- You can set dynamic properties on the KeyViewFilterServiceImpl and KeyViewExportServiceImpl controller services, to modify settings in the KeyView `formats.ini` or `formats_e.ini` configuration file. For example, you can configure the reader to use to process each file format.
- The Eduction processor has an improved guided setup wizard that makes it easier to select entities.
- The NISTRDSFilter processor has been updated to support the NIST RDSv3 hash sets.

### Resolved Issues

There were no resolved issues in this release.

## 23.3.0

### New Features

- NiFi Ingest can enrich IDOL documents by adding more information about a user. There are two new processors, named EnrichUserFromAzureAD and EnrichUserFromLDAP. These query Azure AD or an LDAP directory to obtain information about a user, and add the information to the IDOL document. The processors can read a user ID or e-mail address from either a FlowFile attribute or an IDOL document metadata field. You might use one of these processors when a connector writes a user ID to an IDOL document, and you want to add more information about the user, such as their display name.
- The Eduction processor can extract entities from tables (such as comma-separated or tab-separated data, or the output from KeyView filtering when you set "Output Table Info" to `TRUE`), using information in the table headers to provide context. For more information about the supported table formats, refer to the Eduction documentation.
- The Universal Redaction template supports tables detected by OCR.
- IDOL NiFi connectors support a new advanced property named `adv:AllowedClusterNodesRegex`. By default, a connector runs on all nodes of a cluster. If you set this property the connector runs only on nodes where the hostname matches the specified regular expression.

### Resolved Issues

- NiFi could terminate unexpectedly when a NiFi Ingest media processor (for example the OpticalCharacterRecognition processor) was used to process an image with a very large number of pixels.

- The Content component NiFi processor, `ContentServiceImpl` was unable to obtain a license correctly.

### Notes

- When you use the ContentFromHTML or RenderHTML processors, the embedded web browser is no longer permitted to navigate away from the source page. If necessary you can allow navigation by setting the new configuration parameter `AllowNavigations=TRUE`.

## 23.2.0

### New Features

- The ConvertXMLToDocuments processor has a guided setup wizard.
- The performance of external ODBC datastores has been improved (connectors store their state information in an external database when you set the State Database Service property).

### Resolved Issues

- Lower than expected performance was observed when using an external document registry database. This could result in slow indexing, for example with the PutIDOL processor. The performance of the document registry has been improved.
- The KeyViewFilterDocument processor could fail to process FlowFiles, reporting the error "An invalid or illegal xml character is specified".
- The RemoveDocumentPart processor did not delete temporary .MSG or .EML files if it was used in a dataflow following a KeyViewExtractFiles processor with the "Merge mails" property set to TRUE.
- Reduced memory usage for some IDOL NiFi processors.
- The OpticalCharacterRecognition processor could terminate unexpectedly.
- When using the OpticalCharacterRecognition processor, some temporary files produced by KeyView were written to the system temporary directory, rather than the location specified in the KeyView Export Service. Some temporary files were not deleted when no longer required.

### Notes

- The minimum version of Apache NiFi supported by the IDOL NiFi Ingest components is now NiFi 1.15.3.
- Due to an issue with the storage of OAuth tokens in previous versions of the RMSDecrypt and RMSEncrypt processors, you might need to obtain OAuth tokens again.

## Deprecated Features

The following features are deprecated and might be removed in a future release.

Category | Deprecated Feature | Deprecated Since
--- | --- | ---
Clipping | In the `ContentFromHTML` and `RenderHTML` processors, the `SMARTPRINT` clipping mode has been deprecated. | 23.3.0

Copyright 2025 Open Text
Version 25.4.0 | Last updated October 2025

Send documentation feedback

Close

Send documentation feedback

To send feedback using your default email client, open an email window.

Otherwise, send your feedback and the information below to MFI-swpdl.idoldocsfeedback@opentext.com.

We appreciate your feedback!

Product: Knowledge Discovery 25.4.0

Topic Title: NiFi Ingest