API docs by Redocly

EAIWS Crawler Plugin (1.8)

Download OpenAPI specification:

E-mail: support@easterngraphics.com

Overview

This EAIWS plugin serves to mass collect the information contained in the OFML data. For this purpose, the OFML data is executed and processed via the EAIWS API and then the selected data is exported as an SQLite database. This database can then serve as the foundation for further data analysis and processing.

What can I use this plugin for?

Testing OFML Data

The EAIWS Crawler Plugin offers several possibilities to test the processed OFML data. Errors that occur during processing and performance data are recorded. In addition, there are built-in tests that compare the behavior of the data against a particular expectation.

The scope of possible tests and details of their use are described in the section features for data testing.

Comparing EAIWS releases

Different EAIWS versions can be compared by indexing an identical OFML data set with the EAIWS Crawler Plugin. The databases generated by this process provide information about deviations in the processing of the OFML data or image generation.

Exporting data for further use

The EAIWS Crawler Plugin also creates a simple basis for making the information contained in the OFML accessible to third party applications. For example, this can be used to transfer an article list (including prices and images) into an existing shopping system and to synchronize them on a regular basis. Information about the database are described in the section structure of the database.

How does the processing work?

The most important thing to know about data processing is that it is an asynchronous task that is performed in the background. Depending on the amount of data and features enabled, processing can take from a few seconds to several hours.

Furthermore, the processing is not limited to only one task, but several tasks can be created in parallel and these are also processed in parallel.

When the task is started, the client receives a job id with which it can then query the status of the processing or the result.

For more information, see the REST API section.

Where does the data come from?

First and foremost, the data comes from the OFML data installed locally on the system and accessible to the EAIWS. However, the data used for the actual crawling can be defined in different ways. For this purpose, the EAIWS Crawling Plugin uses the concept of data sources. These limit the data available from the system to a level adapted to the task.

The following data sources are available for the various use cases:

Catalog based data source

This is the simplest data source. It uses the catalog structure stored in the OFML data and indexes all contained articles. The usage of this data source does not require any additional parameters.

OBK based data source

This data source limits the data to be indexed to the content of an OBK. For this purpose, the articles to be indexed are created exactly as they would be used when loading the OBK in one of the pCon applications. To use this data source, an OBK must be provided via a publicly accessible URL. Furthermore, the value for the property type of dataSource must be set to obk.

pCon.ui compatible articles list

This data source allows you to define your own article list based on the information that are used when integrating pCon.ui.

The information describing an article looks like this:

Array
moc
required
string

The manufacturer id or the catalog id of the article.

ban
string

The OFML base article number of the article.

sid
string

The OFML series id of the article.

ovc
string

The OFML variant code of the article.

[
  • {
    }
]

Catalog node key based data source

This data source is also based on the catalog structure present in the OFML data. However, the catalog structure is not crawled. Instead, articles can be referenced and indexed using the unique catalogNodeKey provided by the catalog structure.

Warning: the value for catalogNodeKey is only unique for Article entries across different exports. The value for Folder entries and possibly other types can change each time the OFML data is modified.

Catalog search based data source

This data source uses the catalog search to index the entries.

Note: it is possible that the search result includes entries that are not visible in the normal catalog structure.

Property permutation based data source

This data source uses a OBX and a list of properties to index the entries.

An article is created for all specified properties in all possible variants.

Warning: the number of possible variants is exponential to the selected properties and the number of their values.

Setup

Since it is an EAIWS plugin the setup is quite simple. The plugin only needs to be copied into the EAIWS plugin directory. Otherwise there are no further requirements for the server.

Read and write permissions are the same as for the EAIWS.

Directories

All paths specified are relative to the root folder of the EAIWS installation you are using!

path description
etc/plugins/crawler/worker_config/ This folder contains the configuration files that describe the functionality and crawling behavior. A detailed description of this configuration can be found in the section worker config.
var/plugins/crawler/ In this directory the plugin stores the databases created during crawling. Each file is the result of one crawling task.

Plugin config

This configuration file contains all the settings that should always apply regardless of the crawling task.

The path of this file: etc/plugins/crawler/config.xml

An example could look like this:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">
<properties>
    <entry key="maxBasketWorkersPerJob">8</entry>
    <entry key="maxGlobalConcurrentRunningBasketWorkers">32</entry>
</properties>

List of possible settings

callbackConnectionRetries
integer
Default: 2

The number of retries in case the connection could not be established.

Note: the minimum valid value is 0 and the maximum valid value is 10.

callbackConnectionTimeout
string
Default: "PT5S"

The timeout in milliseconds used for the callback connection.

Note: the maximum valid value is 1 minute.

callbackDelayBeforeRetry
string
Default: "PT1M"

The waiting time between two connection attempts.

Note: the maximum valid value is one day.

callbackReadTimeout
string
Default: "PT1M"

The timeout used for reading the response headers.

Note: the maximum valid value is one hour.

databaseCleanupInterval
string
Default: "PT1D"

The time between the cleanup runs.

Note: the minimum valid value is 1 minute.

databaseMaxAge
string
Default: "PT7D"

The maximum age of a database file. If a database file is older, it will be removed during the next cleanup run.

Note: the minimum valid value is 1 minute.

databaseMaxSize
integer <size in KiB>
Default: 0

This value determines the maximum allowed size for any given database.

During the crawling this is checked regularly and if the value is exceeded the crawling is stopped and the corresponding database is deleted.

Note: the value must be provided as KiB; 0 for unlimited

databaseNameForJournal
string
Default: "journal.db"

The file name of the journal database.

databaseNameFormat
string
Default: "yyyy-MM-dd-HH-mm-ss'.db'"

A format string used to generate the filename of the SQLite database.

The value must be a valid SimpleDateFormat string.

maxBasketWorkersPerJob
integer
Default: 8

This value determines the maximum number of workers used to process the item data per crawling tasks.

maxGlobalConcurrentRunningBasketWorkers
integer
Default: 32

This value determines the maximum number of workers used to process item data across all crawling tasks.

minFreeSpace
integer
Default: 10485760

This value specifies the minimum amount of space that must be available before a crawling operation can be started or continued.

During the crawling this is checked regularly and if the available space falls below the specified value the crawling is stopped and the corresponding database is deleted.

Note: the value must be provided as KiB

spacePatrolInterval
string
Default: "P1M"

The interval at which the size of the active databases and the available space should be checked.

{
  • "callbackConnectionRetries": 2,
  • "callbackConnectionTimeout": "PT5S",
  • "callbackDelayBeforeRetry": "PT1M",
  • "callbackReadTimeout": "PT1M",
  • "databaseCleanupInterval": "PT1D",
  • "databaseMaxAge": "PT7D",
  • "databaseMaxSize": 0,
  • "databaseNameForJournal": "journal.db",
  • "databaseNameFormat": "yyyy-MM-dd-HH-mm-ss'.db'",
  • "maxBasketWorkersPerJob": 8,
  • "maxGlobalConcurrentRunningBasketWorkers": 32,
  • "minFreeSpace": 10485760,
  • "spacePatrolInterval": "P1M"
}

Worker config

This configuration file is an XML file that describes the enabled features and the behavior of a crawling task. In this context, the file name has a special role. It consists of any file name permitted for the operating system followed by the file extension .xml. The file name itself must then be referenced when the crawling is started. (see section REST API)

The structure of the xml corresponds to the format used when defining java properties as xml.

An example could look like this:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">
<properties>
    <entry key="article">true</entry>
</properties>

List of possible settings

"simple" (string) or "deterministic" (string)
Default: "simple"

This value describes how the article indexing tasks should be distributed to the individual workers.

One of
string ("simple")
Default: "simple"

The article are simply distributed to a free worker without any further logic.

insert_ofml_option_no_complex_type
boolean
Default: false

With this option it is possible to disable the processing of complex types (e.g. meta types) when creating articles. For detailed information, please refer to the EAIWS specification.

use_new_fapish_for_each_article
boolean
Default: false

If this option is enabled, a new fapish instance is created for each individual item. This slows down the processing of the OFML data considerably!

This function is used to ensure that the result of the article generation is not dependent on any previously inserted articles.

If the option is disabled, in rare cases of incorrect OFML data it can lead to inconsistent results depending on the order in which the articles were processed within a single fapish instance.

log_level
String
Enum: "Fatal" "Error" "Warning" "Notice" "Info" "Config" "Debug"

This value defines the log level set for processing the data. This has a direct impact on the data contained in the log_entry table. The exact explanation of the different values can be found in the EAIWS specification.

log_stack_trace
boolean
Default: false

If this value is enabled, java stacktrace information are added to the log entries in case of errors in the EAIWS Crawler Plugin or the EAIWS.

catalog_node
boolean
Default: true

This value determines whether the catalog entries should be indexed. If this is disabled, all options starting with catalog_node_ are forcibly disabled.

catalog_node_resources
boolean
Default: false

This value defines whether resources linked in the catalog should be added to the database. This can be for example PDF files or static html pages.

catalog_node_text
boolean
Default: false

This value defines whether additional texts present at a catalog entry should be indexed.

catalog_node_translation
boolean
Default: false

This value defines if the translation of a catalog entry should be indexed. This may also contain a language dependent image.

catalog_node_images
boolean
Default: false

This value defines whether the images contained in a catalog entry should be added to the database.

catalog_node_images_parse_for_correct_dimensions
boolean
Default: false

In some cases, the image dimension stored in the OFML data does not correspond to the actual image dimension. If this value is activated, the actual image content is parsed and the correct dimensions are written into the parsedHeight and parsedWidth fields of the catalog_node_image table.

catalog_package
boolean
Default: false

If this value is activated, information about the catalog packages used during data processing are written to the database.

article
boolean
Default: false

Only when this value is activated, the actual articles are created from the OFML data and the corresponding information is added to the database. If this is disabled, all options starting with article_ are forcibly disabled.

article_obx
boolean
Default: false

This value specifies whether an OBX should be exported after the article creation and added to the database.

article_export
boolean
Default: false

This value specifies whether the client is allowed to request CAD exports and add them to the database.

article_images
boolean
Default: false

This value specifies whether the client is allowed to request article renderings and add them to the database.

article_price_information
boolean
Default: false

This value specifies whether the client is allowed to request article price information and add them to the database.

article_text
boolean
Default: false

This value specifies whether the article texts should be added to the database.

article_copy_paste_test
boolean
Default: false

This value specifies whether an article test should be run which checks if the article can be created via OBX import. This function can be used to find errors in the OFML data.

article_property
boolean
Default: false

This value specifies whether the property list of the article should be indexed.

article_property_value
boolean
Default: false

This value specifies whether every possible value for every single article property of the currently configured article should be indexed.

debug_fapish_stat
boolean
Default: false

This value specifies whether information about the fapish process should be collected during crawling.

Warning: this feature works only on linux and requires access to the /proc file system

debug_fapish_stat_dump
boolean
Default: false

This value specifies whether all information accessible via /proc/[pid]/stat of the fapish process should be saved as json dump.

Warning: this feature requires debug_fapish_stat to be enabled

post_processing_drop_tables
Array of strings (SQLite table name)

This option allows to delete the specified tables from the database after crawling is finished.

{
  • "basket_task_scheduler": "simple",
  • "insert_ofml_option_no_complex_type": false,
  • "use_new_fapish_for_each_article": false,
  • "log_level": "Fatal",
  • "log_stack_trace": false,
  • "catalog_node": true,
  • "catalog_node_resources": false,
  • "catalog_node_text": false,
  • "catalog_node_translation": false,
  • "catalog_node_images": false,
  • "catalog_node_images_parse_for_correct_dimensions": false,
  • "catalog_package": false,
  • "article": false,
  • "article_obx": false,
  • "article_export": false,
  • "article_images": false,
  • "article_price_information": false,
  • "article_text": false,
  • "article_copy_paste_test": false,
  • "article_property": false,
  • "article_property_value": false,
  • "debug_fapish_stat": false,
  • "debug_fapish_stat_dump": false,
  • "post_processing_drop_tables": [
    ]
}

Journal file

This database stores information about all crawling tasks performed. It is used to enable the progress request after the crawling task is finished and to detect if old databases can be deleted.

The name of this can be influenced via the value databaseNameForJournal setting in the plugin config.

Note: this database is purely for the internal processing of the plugin. However, it may be useful for debugging.

crawling_task

This table contains information about the finished crawling tasks.

id
string

The unique job id of the crawling task.

filename
string

The name of the database file.

startTime
integer

The unix time stamp when the crawling was started.

endTime
integer

The unix time stamp when the crawling was finished.

finaleProgressState
string

The final progress state after the crawling is finished.

deleted
integer [ 0 .. 1 ]

If the field is set to 1, the database file has already been deleted.

{
  • "id": "string",
  • "filename": "string",
  • "startTime": 0,
  • "endTime": 0,
  • "finaleProgressState": "string",
  • "deleted": 1
}

REST API

The plugin provides its functions via REST interface.
Below is a list and description of the API endpoints.

Start the crawling process

This request starts the crawling process. In case of success a jobId is delivered as a response.
However, the actual processing of the job can take from a few seconds to several hours.

The status and the final result can be requested using the other API endpoints.

Request Body schema: application/json
workerConfig
required
string

Name of the worker config to be used.
For more information, please refer to the section worker config.

Catalog based data source (object) or OBK based data source (object) or pCon.ui compatible articles list (object) or Catalog node key based data source (object) or Search based data source (object) or Property permutation (object)

The data source can be used to influence the data which will be indexed.

For more information, please refer to the section where does the data come from.

Note: if no value is provided, the Catalog based data source is used.

One of
type
required
string
Value: "catalog"

Name of the data source to be used.

object
displayMode
string
Default: "All"
Enum: "All" "Planning2D" "Planning3D" "Configuration" "CAD" "AllVisible" "AllVisibleBasket"

The DisplayMode used for the catalog.

Note: Detailed information on the meaning of the values can be found in the EAIWS specification.

Array of objects

The list of arguments used for creating the EAIWS session.

More detailed information can be found in the EAIWS spec.

Array
first
string
second
string
callbackUrl
string <uri>

This URL will be called by the plugin after the crawling finished.

More details can be found in the callback section below.

clientId
string

The clientId is always needed if the package groups to be used are to be obtained via pCon.update.

For more information, please refer to the pCon.login documentation.

accessToken
string

A short-lived pCon.login access token. This can be used to query the packet groups via pCon.update.

For more information, please refer to the pCon.login documentation.

refreshToken
string

A long-lived pCon.login refresh token. This can be used to query the packet groups via pCon.update.

For more information, please refer to the pCon.login documentation.

priceDate
string <date>

The value to be set as the price date. If no value is specified, the current date is used.

languageList
Array of strings <ISO 639 alpha-2 language code> [ items <ISO 639 alpha-2 language code > ]

A list of ISO 639 alpha-2 language codes.

The specified languages are used for exporting the properties depending on specific languages.

disable
Array of strings

This list can be used to disable features which are enable in the specified worker config.

object

This mapping is used when generating CAD exports.

The key corresponds to the name of the export to be stored in the key field of the article_export table. The value must contain a valid list of CAD export options.

export key*
additional property
Array of strings <key=value> [ items <key=value > ]

List of CAD export settings.

For more information, please refer to the EAIWS documentation.

object

This mapping is used when rendering article images.

The key corresponds to the name of the rendering to be stored in the key field of the article_image table. The value must contain a valid list of rendering settings.

rendering key*
additional property
Array of strings <key=value> [ items <key=value > ]

List of rendering settings.

For more information, please refer to the EAIWS documentation.

basketWorkers
integer

This value determines the number of workers to be used for processing the article data.

Note: the value cannot exceed the maximum value from the worker config.

Array of ItemAppData (object) or DeterministicOBXIds (object)

Hooks are used to influence the behavior of the crawling or to extend the data to be stored.

Array
One of
type
required
string
Value: "ItemAppData"

This hook makes it possible to store additional application data on the processed item. This meta data is then included, for example, in some exports such as the OBX.

required
object
mode
required
string
Enum: "xml" "text"

Determines the way in which the data is written:

mode action
xml The value is parsed as xml and child elements are created accordingly.
text The value is treated as simple text and saved directly as the content of the application node.
appKey
required
string

The appKey used for storing the value.

value
required
string

The value to be stored.

debugFilterCatalogIds
Array of strings (OFML catalog id)

This is a debug function that will filter the list of catalog entries present in the data using the specified list of catalog ids.

debugIndexOnlyEveryNthArticle
integer

This is a debug function that can be used to limit the number of articles that are indexed. Only every nth entry will be indexed. Where n is the value set by this parameter.

debugMaxDataSourceEntries
integer

This is a debug function that can be used to limit the number of articles that are indexed. The indexing is stopped as soon as the number of articles specified by this value has been processed.

Responses

Response Schema: application/json
jobId
required
string

The unique id assigned to a crawling job.

Callbacks

Request samples

Content type
application/json
{
  • "workerConfig": "string",
  • "dataSource": {
    },
  • "openSessionArguments": [
    ],
  • "callbackUrl": "http://example.com",
  • "clientId": "string",
  • "accessToken": "string",
  • "refreshToken": "string",
  • "priceDate": "2025-01-31",
  • "languageList": [
    ],
  • "disable": [
    ],
  • "geometryExportSettings": {
    },
  • "imageExportSettings": {
    },
  • "basketWorkers": 0,
  • "hooks": [
    ],
  • "debugFilterCatalogIds": [
    ],
  • "debugIndexOnlyEveryNthArticle": 0,
  • "debugMaxDataSourceEntries": 0
}

Response samples

Content type
application/json
{
  • "jobId": "string"
}

Callback payload samples

Callback
POST: Progress notification
Content type
application/json
{
  • "jobId": "string",
  • "successful": true
}

Cancel the crawling process

This request can be used to cancel a still running crawling job.

path Parameters
jobId
required
any

The unique id assigned to a crawling job.

Request Body schema: application/json
deleteDatabase
boolean

If set to true, the database file will be deleted immediately.

nonBlocking
boolean

If set to true, the request is responded to before the crawling process and all associated threads or workers have completed their execution.

Responses

After successful abort of crawling, the crawling status reached up to this point is returned.

Response Schema: application/json
finished
required
boolean

If set to true all crawling tasks including post processing are finished.

Note: the value true only indicates that crawling task was finished. However, it was not necessarily successful. Therefore the cancelled property must also be evaluated.

cancelled
required
boolean

If set to true then the crawling was cancelled and the database is most likely incomplete.

cancelReason
string

A message describing why the crawling was cancelled.

counting
required
boolean

Only when this is set to true, the final values for the total property of the items entries are determined. Before this is the case, these can change at any time or new items can be added.

required
Array of objects

This list contains a detailed overview of the found entries and their types as well as status information about their processing.

Array
type
required
string

The type of the entry. For example, article or folder.

processed
required
string

The number of the already processed entries.

total
required
string

The total number of entries found.

Request samples

Content type
application/json
{
  • "deleteDatabase": true,
  • "nonBlocking": true
}

Response samples

Content type
application/json
{
  • "finished": true,
  • "cancelled": true,
  • "cancelReason": "string",
  • "counting": true,
  • "items": [
    ]
}

Get the crawling progress

This request can be used to retrieve crawling progress.

path Parameters
jobId
required
any

The unique id assigned to a crawling job.

Responses

Response Schema: application/json
finished
required
boolean

If set to true all crawling tasks including post processing are finished.

Note: the value true only indicates that crawling task was finished. However, it was not necessarily successful. Therefore the cancelled property must also be evaluated.

cancelled
required
boolean

If set to true then the crawling was cancelled and the database is most likely incomplete.

cancelReason
string

A message describing why the crawling was cancelled.

counting
required
boolean

Only when this is set to true, the final values for the total property of the items entries are determined. Before this is the case, these can change at any time or new items can be added.

required
Array of objects

This list contains a detailed overview of the found entries and their types as well as status information about their processing.

Array
type
required
string

The type of the entry. For example, article or folder.

processed
required
string

The number of the already processed entries.

total
required
string

The total number of entries found.

Response samples

Content type
application/json
{
  • "finished": true,
  • "cancelled": true,
  • "cancelReason": "string",
  • "counting": true,
  • "items": [
    ]
}

Get the crawling results

This request can be used to retrieve the actual crawling result.

Note: the crawling must be finished before the database can be retrieved.

path Parameters
jobId
required
any

The unique id assigned to a crawling job.

Responses

Response Schema: application/octet-stream
string <binary>

This response returns the generated SQLite database as a binary data.

Structure of the database

Things to be aware of

  • The tables present in the database depends on the worker config and the request settings.
  • The database has no indexes except the primary keys. Since an index usually has to be tailored to a specific use case and an unnecessary index would negatively influence the size of the database and the time needed for the creation, no additional indexes are created.
  • In some cases, lists of strings are written to a database field as a JSON array. For the fields in question, separating out the information in an extra table only makes sense in a few rare use cases. Therefore, it was not used in favor of reducing the size of the database. In addition, SQLite supports the processing of JSON data by default.

Relationship between tables

All tables have a field named parentId. This field points to the record with the id of the parent table.

The tables of the highest order (e.g. article or catalog) get the value of the parentId from the data source.

Some tables like meta or catalog_package are completely independent. In such cases these will be explained separately in the table description.

The following is a graphical illustration of the relationships between the tables and the data source:

table overview

Tables

article

This table contains the basic information about the articles generated from the OFML data.

parentId
string

A unique id generated by the data source.

packageName
string

The OFML name of the package containing the OFML article data last used to insert or configure the OFML article.

progId
string

The OFML program id of the package containing the OFML article data last used to insert or configure the OFML article.

distributionRegion
string

The OFML distribution region of the package containing the OFML article data last used to insert or configure the OFML article.

baseArticleNumber
string

This field contains the base article number.

finalArticleNumber
string

This field contains the final article number.

ofmlVariantCode
string

This field contains the OFML variant code of the OFML article. The OFML variant code always uses a key-value encoding, allowing it to be reliably parsed. This element is returned for OFML articles only.

manufacturerId
string

This field contains the commercial identifier of the article\u2019s manufacturer.

seriesId
string

This field contains the identifier for the article\u2019s series.

hasInconsistency
integer [ 0 .. 1 ]

The value of this field indicates whether or not the EAIWS detected an inconsistency in the article\u2019s current configuration. If this field is 1, an inconsistency has been detected, and the article\u2019s price and/or the position\u2019s value are probably invalid.

geometryChecksum
string

The value of this field contains a checksum based on the information relevant for the CAD exports.

Note: The following flags are set for the calculation of the checksum:

  • OnlyIfKnown
  • IncludeSubArticles
  • Include2DSymbols
  • WithTransformation

Their meaning can be found in the EAIWS documentation.

obx
string <binary>

The content of the exported OBX file.

Note: requires that the article_obx feature is enabled

insertTime
integer <time in milliseconds>

The time required for the article creation.

sessionId
string

The sessionId used for indexing this article.

Warning: this field is only available if the corresponding debug feature was activated when building the plugin

article_copy_paste_test

This table contains the result of the copy paste test. For this purpose the successfully created article is exported as OBX. After that, we create a new article based on the OBX.

This test is used to find errors in the OFML data.

parentId
string

The same parentId that is used in the article table.

failed
integer [ 0 .. 1 ]

If the value is set to 1, the OFML article data contains an error that prevents the article from being created from an OBX.

article_export

This table contains the CAD exports.

parentId
string

The same parentId that is used in the article table.

key
string

The key corresponds to the key that was defined during the crawling request.

data
string <binary>

The binary data of the generated export.

exportTime
integer <time in milliseconds>

The time required for the export.

article_image

This table contains the article rendering.

parentId
string

The same parentId that is used in the article table.

key
string

The key corresponds to the key that was defined during the crawling request.

image
string <binary>

The binary data of the generated rendering.

renderTime
integer <time in milliseconds>

The time required for the rendering.

article_price_information

This table contains information about the article prices stored inside the OFML data.

Note: all values of these entries come directly from the article data. No calculation via calculation schema is used.

parentId
string

The same parentId that is used in the article table.

distributionRegion
string

The distribution region for which the prices are valid.

currencyCode
string <ISO 4217>

If the article position has a price, then this field contains the ISO 4217 currency code of the price.

pdPurchasePrice
number <float>

This field contains the purchase price for the current configuration of the article as determined by the product data. If no purchase price could be determined, the field is not available. The field will not be provided if the currencies of purchase price and sales price are different and the value of the purchase price is non-zero.

pdSalesPrice
number <float>

This field contains the sales price for the current configuration of the article as determined by the product data. If no sales price could be determined, the field is not available.

purchasePrice
number <float>

This field contains the purchase price as read from the product database, possibly adjusted by an OFML price profile. If no purchase price could be determined, the field is not available. The field will not be provided if the currencies of purchase price and sales price are different and the value of the purchase price is non-zero.

salesPrice
number <float>

This field contains the sales price as read from the product database, possibly adjusted by an OFML price profile. If no sales price could be determined, the field is not available.

salesUnitGrossWeight
number <float>

The gross weight of the sales unit.

salesUnitNetWeight
number <float>

The net weight of the sales unit.

salesUnitSize
number <float>

The size of the sales unit.

salesUnitVolume
number <float>

The volume of the sales unit.

isDiscountable
integer [ 0 .. 1 ]

If the value is set to 1, the item is discountable.

article_property

The entries of this table correspond to basket item properties specific to article items.

id
integer

A unique id for this entry.

This value is used for mapping between this and the article_property_value table.

parentId
string

The same parentId that is used in the article table.

languageTag
string

The language used while processing the data.

class
string

The property class is a symbolic name used to classify properties. Property class and property name uniquely identify the property with respect to the article. The property class names stored in this field correspond to the property class name stored in the name field of the PropertyClass structure.

classText
string

The property classText contains the textual description of the property class.

name
string

The property name is the symbolic name of the property. The property name is unique relative to the property class, but there is no guarantee that no two properties of an article have the same property name.

type
string

This is the fundamental type of the property.

More detailed information can be found in the EAIWS spec.

text
string

The property text is a short human readable description of the property, suitable for use in a tabular property editor. The language of the text depends on the configuration of the current session and the languages available in the product data.

width
integer

This is the maximum number of characters for properties of type Character. For other property types the value of this field is undefined and should not be used.

The value of this field will never change for a particular property.

digits
integer

This is the maximum number of significant decimal digits for properties of type Numeric and Length. For other property types the value of this field is undefined and should not be used.

The value of this field will never change for a particular property.

decDigits
integer

For properties of type Number and Length, this is the number of decimal digits right of the decimal point (the precision). For other property types the value of this field is undefined and should not be used.

The value of this field will never change for a particular property.

visible
integer [ 0 .. 1 ]

This field indicates whether the property should be displayed by a property editor.

The value of this field may change depending on the current configuration.

editable
integer [ 0 .. 1 ]

This field indicates whether the property may be changed by the client. It should be ignored for invisible properties. The value of this field may change depending on the current configuration.

The field editable is always true if field visible is false. The OFML API does not allow to differentiate between editable and read-only invisible properties, and EAIWS does not prevent the client from setting the value of an invisible property, so the value true is more appropriate.

addValues
integer [ 0 .. 1 ]

This field indicates whether the property may assume values (and may be set by the client to a value) other then the values found in the choice list of the property. For numeric and length properties, the restrictions imposed by possible intervals are not affected by the value of this field.

choiceList
integer [ 0 .. 1 ]

This field indicates whether a property has a choice list. Unless the field addValues is 1 a choice list restricts the allowed property values to the elements of the choice list. The value of this field may change depending on the current configuration.

valueValue
string

This field contains information about the current internal value of the property.

valueText
string

This field contains information about the current human readable value of the property.

valueSmallIcon
string <binary>

If the property has a choice list, and a small icon is available for the current property value, then this field contains the image for the small icon.

valueLargeIcon
string <binary>

If the property has a choice list, and a large icon is available for the current property value, then this field contains the image for the large icon.

article_property_value

The entries of this table describe the individual values of an item property.

parentId
integer

A reference to the id of the corresponding entry from the article_property table.

value
string

This is the internal property value. For properties of type Character this field contains either an internal symbolic character sequence or human readable text. For properties of type Numeric and Length this field contains the numeric value of the property formatted as follows: The integral part of the value is formatted as an optional minus sign followed by one or more decimal digits without any insignificant leading zeros. If the property is declared with a non-zero number of decimal digits (digits right of the decimal point), the formatted integral part is followed by a decimal point (.) and the number of declared decimal digits. XXX

text
string

This is the human readable property value. For properties of type Character this is either the same as value, or, if value is a symbolic character sequence, a human readable text representing the symbolic value. The language used for the human readable text depends on the configuration of the current session and the languages supported by the product data. For properties of type Numeric and Length this is the numeric value of the property formatted according to the rules of the locale configured for the current session.

smallIcon
string <binary>

If the property has a choice list, and a small icon is available for this property value, then this field contains the image for the small icon.

largeIcon
string <binary>

If the property has a choice list, and a large icon is available for this property value, then this field contains the image for the large icon.

article_text

This table contains the language dependent attributes of an article.

parentId
string

The same parentId that is used in the article table.

language
string

The language used while processing the data.

shortText
string

This field contains the short description of the article. While not enforced by the EAIWS, the short description should consist of a single line of text only. The short description describes the article and is supposed not to depend on the current configuration of the article.

longText
string

This field contains the long description of the article, possibly consisting of multiple lines of text. The long description may describes the article in more detail than the short description, but, like the short description, should not depend on the current configuration of the article.

featuresText
string

The feature description describes the configuration-dependent features of the article.

catalog

This table contains entries for each manufacturer catalog used for crawling.

catalogId
string

This field uniquely identifies the catalog within one instance of the EAIWS.

More detailed information can be found in the EAIWS spec.

releaseDate
string

This field contains the release date of the catalog package.

distributionRegion
string
priceProfileRegionId
string
manufacturerIds
string <JSON array of strings>

This field contains the list of the manufacturerId included in the catalog structure.

catalog_node

This table contains the language independent information of the OFML catalog entries.

parentId
string

A unique id generated by the data source.

catalogId
string

This field uniquely identifies the catalog within one instance of the EAIWS.

More detailed information can be found in the EAIWS spec.

catalogPackageId
string

This field contains the package ID of the OFML package whose catalog data contains the catalog node.

itemType
string

This field contains the type of the catalog node.

More detailed information can be found in the EAIWS spec.

path
string <JSON array of strings>

This field contains the breadcrumb path for the catalog entry.

articleNumber
string
articlePackageId
string
variantCode
string

This field contains the variant code which should be used when creating the article.

varCodeType
string

This field contains the type of the variant code.

catalog_node_image

This table contains the images referenced by the catalog_node entries.

parentId
string

The same parentId that is used in the catalog_node table.

image
string <binary>

This field contains the actual content of the referenced file.

size
integer

This field contains the file size of the referenced file.

extension
string

This field contains the file extension of the referenced file.

width
integer

This field contains the image width as specified in the OFML data.

height
integer

This field contains the image height as specified in the OFML data.

parsedWidth
integer

This field contains the real width of the image determined by a function which parses the actual image data.

parsedHeight
integer

This field contains the real height of the image determined by a function which parses the actual image data.

highRes
integer [ 0 .. 1 ]

The value of the high-resolution flag as specified in the catalog data.

Note: In case of an XCF catalog, the value of highRes is always 0.

catalog_node_resource

This table contains resource entries which are referenced by the catalog entry.

Common entries are e.g. PDFs linked in the catalog or other static resources.

parentId
string

This field contains the same parentId that is used in the catalog_node table.

languageTag
string

This field contains the language as specified in the catalog data.

type
string

This field contains the resource type.

More detailed information can be found in the EAIWS spec.

name
string

This field contains the name of the resource.

mime
string

This field contains the mime type of the resource file.

content
string <binary>

This field contains the content of the resource file.

catalog_node_text

This table contains all texts referenced by the catalog_node entries.

parentId
string

The same parentId that is used in the catalog_node table.

languageTag
string

The language as specified in the catalog data.

purpose
string

The text purpose.

Predefined values are CatTextShort and CatTextLong. Other values may be reported depending on the catalog data and text purposes specified as part of the lookup options. In case of an XCF catalog, CatTextShort is used as the text purpose for text from the text table.

value
string

The actual text.

catalog_node_translation

This table contains the language dependent attributes of an catalog_node.

parentId
string

The same parentId that is used in the catalog_node table.

languageTag
string

The language used while processing the data.

text
string

This field contains a short text describing the catalog node. It is supposed to be displayed as part of the catalog presented to the user.

icon
string <binary>

The icon field contains the small image that should be displayed as part of the catalog presented to the user.

image
string <binary>

catalog_package

This table information about the DSR packages used for crawling.

id
string
manufacturer
string

This field contains the name of the manufacturer.

manufacturerId
string

This field contains the commercial identifier of the manufacturer.

program
string

This field contains the OFML program set in the catalog package.

concernId
string
productDb
string
productDbPath
string
releaseVersion
string

This field contains the value of DSR key release_version.

releaseDate
string

This field contains the value of DSR key release_date.

releaseTimestamp
string

This field contains the value of DSR key release_timestamp.

dependency
string <JSON array of strings>

This field contains a list of catalog_package on which this package depends.

catalogType
string

This field contains the type of the catalog data.

e.g. XCF or OAS

distributionRegion
string

This field contains the OFML distribution region set in the catalog package.

features
string <JSON array of strings>

This field contains a list of special catalog features.

More detailed information can be found in the EAIWS spec.

languages
string <JSON array of strings>

This field contains a list of the languages used by this package.

metaType
string
priceProfileRegionId
string

This field contains the value of DSR key ppr_region_id.

progInfo
string
releaseState
string

This field contains the value of DSR key release_state.

sellingCurrency
string
seriesIds
string <JSON array of strings>

This field contains the value of DSR key program_id.

specialArticleScheme
string
supplierId
string
versionedName
string
defaultPackage
integer [ 0 .. 1 ]
visible
integer [ 0 .. 1 ]

catalog_package_translation

This table contains the language dependent attributes of an catalog_package.

parentId
string
languageTag
string
manufacturerName
string
manufacturerNameUseManCfg
string
distributorName
string
distributorNameUseManCfg
string
programName
string
description
string
copyright
string
releaseText
string

catalog_package_series

This table contains information about the series.

parentId
string

This field contains the id of the corresponding catalog_package entry.

languageTag
string

This field contains the language used while processing the data.

id
string

This field contains the series id.

name
string

This field contains the series name.

catalog_translation

This table contains the language dependent attributes of an catalog.

parentId
string

This field contains the same parentId that is used in the catalog_node table.

languageTag
string

This field contains the language used while processing the data.

name
string

This field contains the display name of the catalog. It is supposed to be displayed as part of the catalog presented to the user.

icon
string <binary>

This field contains a small image that should be displayed as part of the catalog presented to the user.

iconExtension
string

This field contains the file extension of the images used as icon.

debug_fapish_stat

This table contains information about process statistics of the fapish.

All recorded information originate from the proc file system. Therefore, the corresponding man page (/proc/[pid]/stat) should be read to interpret the values.

parentId
string

The field parentId contains a reference to an entry of the table referenced in the field parentTable.

taskIdentifier
string <JSON array of strings>

A list of strings that uniquely identifies the task previously performed.

rss
integer

The number of pages the process has in real memory.

utime
integer <clock ticks>

Amount of time that the fapish process has been scheduled in user mode, measured in clock ticks.

dump
string

A complete dump of all stat information.

Note:requires that the debug_fapish_stat_dump feature is enabled

log_entry

This table contains information about errors that occurred during data processing. The errors are in most cases related to problems in the OFML data itself.

However, in a rare case, errors in the programming logic of the EAIWS or the EAIWS Crawler Plugin may be recorded.

parentId
string

The field parentId contains a reference to an entry of the table referenced in the field parentTable.

parentTable
string

The field parentTable contains the name of the table which together with the value of the field parentId gives a reference to the record that generated the log entry.

level
string

The filed level indicates the severity of the log entry.

More detailed information can be found in the EAIWS spec.

facility
string

The filed facility contains a sequence of one or more identifiers separated by single or double colon; Usually, the facility specified the module that produced the log message. Special facilities are OFML for OFML-related problems detected by EAIWS, OBX for syntactic errors when reading BSK/OBX streams, and GF::* for messages generated by FAPI-Shell.

message
string

The filed message contains the actual log message.

className
string

The field className contains the name of the java exception if the the log message was caused by an java exception.

stackTrace
string

The field stackTrace contains information about an java exception if the the log message was caused by an java exception.

meta

This table contains meta information about the crawling job. The structure corresponds to a key value map.

The entries may vary depending on the version of the plugin and the configuration of the crawling task. However, the following entries should always be present:

key value
eaiwsVersion The version of the EAIWS used for crawling.
pluginVersion The version of the crawling plugin used for crawling.
startTime The unix time stamp when the crawling was started.
endTime The unix time stamp when the crawling was finished.
maxConcurrentFapish The number of fapish processes used in parallel for crawling the article data.
basketWorkerQueue The name of the queue class used for scheduling crawling tasks between workers.
cancelled Determines if the crawling was cancelled and therefore the database is incomplete.
cancelReason The reason that caused the crawling to be cancelled.
requestConfig The request config used for crawling. Note: the value does not correspond exactly to what was sent in the request body. Sensitive information is omitted and possibly undefined values are replaced by their default values.
key
string

This field contains a unique key that can be used to identify the meta data.

value
string

This field contains the actual value of the meta data.

Features for data testing

Articles that cannot be created

The easiest way to search for errors in the OFML data is to compare the entries from the catalog_node table with the entries from the article table.

For each parentId from the catalog_node table with the itemType of the value Article a corresponding entry with the identical parentId must exist in the article table.

The following SQL query returns a list of catalog_node entries for which the article creation failed.

SELECT
    *
FROM
    catalog_node
WHERE
    itemType = 'Article' AND
    parentId NOT IN (
        SELECT parentId FROM article
    )

Note: Information about the actual errors can be found in the log_entry table.

Articles that cannot be created via OBX

In many applications it is common to save the state of an article as OBX and to restore it's state again at a later time. Since this important function can be affected by errors in the OFML data, an additional test has been added to the crawler plugin.

To activate this test the value for article_copy_paste_test in the worker config used for crawling must be set to true.

This will add the table article_copy_paste_test to the database created during crawling.

Article with broken dependencies

Depending on the use of certain other articles, errors may occur for articles with incorrect information about their dependencies. Such errors can be detected by using a new fapish instance for every single article.

To activate this feature the value for use_new_fapish_for_each_article in the worker config used for crawling must be set to true.

Note: Activating this feature slows down crawling significantly!

Changelog

This section covers only the changes to the API, the configurations or the database. Therefore, the version information in this documentation refers to the major and minor level of the actual plugin versions. The patch level of the plugin version is ignored.

Example: the version 1.0 from this documentation applies to the plugin versions 1.0.0, 1.0.1, 1.0.2 and so on

1.8

API

1.7

Feature summary

  • it is now possible to set and get the priceDate used for crawling
  • it is now possible to index article permutations based on a list of selected properties

API

Database

  • a new entry priceDate has been added to the meta table

1.6

Feature summary

  • it is now possible to store additional application data for each item
  • it is now possible to replace the random ids within an OBX with deterministic ones
  • the memory usage has been reduced by releasing various resources as soon as possible

API

Bug fixes

1.5

Breaking changes

  • EAIWS version 4.16alpha1 or newer is required

Database

  • a new field releaseTimestamp has been added to the catalog_package table
  • a new entry requestConfig has been added to the meta table

Bug fixes

  • fixed a problem when reading resources of type OAS_MIME that caused the corresponding content column of the catalog_node_resource table to always be empty

1.4

Feature summary

  • it is now possible to index catalog nodes based on a search result

API

Database

  • a new field geometryChecksum has been added to the article table

1.3

Database

1.2

Database

1.1

Feature summary

  • it is now possible to record the exact RAM and CPU consumption of the fapish when processing an article
  • it is now possible to disable crawling when the available disk space falls below a threshold value
  • it is now possible to define the maximum allowed size for the crawling database
  • it is now possible to influence crawling in such a way that the articles processed within a single session are deterministic

API

Database

  • a new optional field sessionId has been added to the article table
  • a new field renderTime has been added to the article_image table
  • a new field exportTime has been added to the article_export table
  • the new table debug_fapish_stat has been added
  • the new entries basketWorkerQueue, cancelled and cancelReason have been added to the meta table

Config

  • a new option debug_fapish_stat has been added to the worker config
  • a new option debug_fapish_stat_dump has been added to the worker config
  • a new option databaseMaxSize has been added to the plugin config
  • a new option minFreeSpace has been added to the plugin config
  • a new option spacePatrolInterval has been added to the plugin config

Bug fixes

  • fixed incorrect total counter of the progress response
  • fixed a race condition which causes an invalid reply when canceling a crawling job