Common Issues and Solutions

HPCC4J users should review this list of common issues and solutions before engaging the development team. If a specific issue isn't addressed in this list, feel free to report issues for the development team via our Jira system (Make sure to specify the project name: "JAPI" when creating new Jira issue).

DFSClient read fails: "Read failure for X copy locations"...

DFSClient read fails: "Read failure for X copy locations"... "createRemoteActivity: unathorized"

• DFSClient actions are reliant on properly configured "ClusterGroup" Keys on the target HPCC Systems cluster.
• User should contact the HPCC cluster admin and ensure the Keys section is correctly configured
• Sample Keys configuration element:

<Keys>
    <ClusterGroup keyPairName="mythor" name="mythor"/>
    <ClusterGroup keyPairName="mythor" name="myroxie"/>
    <ClusterGroup keyPairName="mythor" name="hthor__myeclagent"/>```
    <KeyPair name="mythor" privateKey="/home/hpcc/certificate/key.pem" publicKey="/home/hpcc/certificate/public.key.pem"/>
</Keys>

• Common issues include: Invalid keypairs, inconsistent keypairs across all nodes, invalid keypair paths, no keypair associated with target ClusterGroup

Create access denied for scope

'Create access denied for scope' error received from ESP

• This situation usually arises when the caller does not have appropriate file scope access
• Contact the administrator responsible for the target HPCC cluster
• It is possible file permissions are not shared across environments. See conversation in below Jira: https://track.hpccsystems.com/browse/JAPI-456

Code: '8029' Assertion Error

Assertion Errors: Such as: Code: '8029' Message: 'ERROR: cmd=RFCStreamRead, error=Internal Error (3000, assert(required <= maxAvailable()) failed - file: rtlcommon.hpp, line 137) '

• These are generally caused by a mismatch between the provided 'source record layout' and the record layout on disk.
• Note this is the 'source record layout' not the projected record layout

Unable to read next record:

Unable to read next record: java.util.NoSuchElementException: Fatal read error: Failed to parse next record: Error while parsing field: vault_rid of type: INTEGER:

• Generally caused by the socket being closed on the server side, unexpectedly. This could be a network connectivity issue or an issue encountered by dafilesrv.
• Debug by checking dafilesrv logs

Missing records when reading is finished

Missing records when reading is finished

• If an error is encountered during the read process, such as a network connectivity issue, a partial read of a file will occur.
• However, an exception will be thrown when this occurs; if an exception is thrown during the read process of a file that file read should be restarted via HPCCRemoteFileReader's resume support. See: HPCCRemoteFileReader.getFileReadResumeInfo().
• Note: Internally if a connectivity issue occurs the reader will attempt to re-establish a connection 3 times before it gives up and throws an error
• If resuming the read also fails the read records should be discarded and the file part in question should be read from the beginning.

Long type overflow error message

Long type overflow error Message

• HPCC has support for unsigned8 support, but Java does not have a native type that would support that entire range unsigned.
• However, most of the time unsigned8's are used as unique IDs.
• In which case it is ok if those IDs are negative so by default we put unsigned8's in a Java Long, and warn the user when a value overflows and becomes negative.
• There is an option to treat unsigned8's as BigIntegers

Authentication Issue

Authentication Issue: Received ERROR from Thor node (x.y.z.w): Code: '-11' Message: 'ERROR: cmd=RFCStreamRead, error=RFSERR_Unknown (4294967285, createRemoteActivity: unathorized)

• Usually bad user credentials, could also be misconfigured Keys see below

Keys Issue

Keys Issue: EspException: Audience: user Source: CSoapResponseBinding Message: 2020-08-11 17:32:32 GMT: error:00000000:lib(0):func(0):reason(0) - CLoadedPrivateKeyFromFile: failed to open private key Received ERROR from Thor node (10.173.14.200): Code: '-11' Message: 'ERROR: cmd=RFCStreamRead, error=RFSERR_Unknown (4294967285, createRemoteActivity: unathorized)_ : Failure to read hpcc file part>thor_data400::in::uccv2::20190809::dnb::debtor-400 copy locations: {10.173.14.200, 10.173.14.101} :7601

• Generally caused by misconfigured keys in the HPCC config for the Thor cluster begin acccessed

Timeout issue

Timeout issue: java.lang.AssertionError: Failed to write file with error: Row service returned error code: '-7' Message: 'ERROR: cmd=RFCStreamGeneral, error=RFSERR_Unknown (4294967289, createRemoteActivity: authorization expired)

• This is caused by read / write token expiring before it is used.
• Increasing the timeout is not recommended, best solution is to warm start connections to each file part before reading / writing. That way the tokens get used to start the connections before they expire.

SSL Issue

SSL Issue

• If the cluster is using https with a self-signed certificate. Any computers using HPCC4j to communicate with the cluster will need to add the SSL cert to their trusted key store:
• keytool -import -storepass changeit -noprompt -alias hpccsystems -file /home/hpcc/certificate/certificate.pem -keystore /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts

Certificate for doesn't match any of the subject alternative names: [some.domain]

Certificate for <X.Y.Z.V> doesn't match any of the subject alternative names: [some.domain]

• SSL based communication between WsClient/DFSClient/Spark-HPCC and HPCC can be affected by invalid security certificates or mismatched connection target. The error message signifies mismatch between certificate SAN and connection target.
• Ensure the connection target address is covered by the certificate SAN
• This could require the use of appropriate hostname, and/or addition of said host name mapping in certificate SAN section