Home

Sailfish User Manual - Exactpro

1 Introduction

Sailfish is a test automation tool for testing bi-directional message flows in distributed trading platforms and market data delivery systems. The tool interacts with system under test via client-facing and internal protocols, including REST API. The purpose of Sailfish is to minimize manual intervention required to execute test suites. In its more sophisticated deployments, Sailfish makes it possible to achieve fully autonomous ordered test execution that does not require ongoing operator monitoring.

Sailfish has a modular structure, whereby a shared framework is used in conjunction with specialized plug-ins. Separate plug-ins are used for each project and include necessary protocol services, such as industry-standard protocols (e.g. FIX) and proprietary protocols.

2 Prerequisites for Installation

Sailfish is a cross-platform application. This section provides information regarding the recommended configuration of the system to install Sailfish.

2.1 Operating System

The prerequisites for installation are as follows:

1. Windows 32 bit (XP or higher) / Suse Linux (Enterprise Server 11 or higher);

2. User account with admin privileges.

2.2 Software Requirements

1. JDK 1.8u201 or higher

2. Tomcat 9.0.17 or higher (you can download tar.gz distributive from tomcat.apache.org)

3. Supported DBMS

• MySQL server 5.5 or higher (exclude 7 or higher);
• Postgresql 9.1 or higher;
• Any DBMS with supporting JDBC driver and Hibernate Dialect.

2.3 Hardware Requirements

The hardware requirements for successful installation are as follows:

1. Dual core processor 2.5 GHz (Intel, AMD);

2. 4GB RAM (recommended);

3. 10 GB HDD.

2.4 Browser Requirements

Google Chrome 65.0.3325.181 or higher.

3 Installation

Before starting the installation of Sailfish on a computer, make sure that it meets the requirements outlined in section 2. Make sure that MySQL/PostgreSQL and Tomcat servers have been deployed.

3.1 Installation

3.1.1 Linux

1. Create 2 folders on any disc with a sufficient amount of space: Sailfish/ - it will contain the application itself, and tmp/ – a temporary folder for archives.

2. Download the Tomcat server (apache-tomcat-8.x.xx or higher, e.g.: http://mirror.linux-ia64.org/apache/tomcat/tomcat-9/v9.0.12/bin/ pache-tomcat-9.0.12.tar.gz).

3. Unpack it to the Sailfish folder created in step 1 above.

NOTE!: Please note that Apache Tomcat is not a part of the Sailfish installation archive and should be installed separately (https://tomcat.apache.org/download-90.cgi).

4. Download a set of Sailfish installation files – the core distributive for Sailfish and the required plug-ins. (Alternatively, you can request them from GitHub https://github.com/Exactpro/sailfish-core).

Usually, the archive with the core set of files contains the following: DB, Tomcat, Workspace folders and a file named sfgui.war.

5. Unzip the archive with the core set of files to the tmp/ folder created in step 1 above.

6. Create folder in <Deployed Tomcat>/webapps/demogui.

7. Extract the war archive from <tmp>/sfgui.war file to the <Deployed Tomcat>/webapps/demogui folder.

8. Extract the zipped archive with Sailfish plugins to the <Deployed Tomcat>/webapps/demogui folder.

9. The <tmp>/DB folder contains different scripts for creating databases. There are ready-made sets of scripts for MySQL and PostgreSQL.

10. For example, you can choose the create_mysql_db.sh file to define the correct parameters of the database. It may also be required to set up a “superuser” password. You can pass the required options as script arguments (create_mysql_db.sh -superpaassword <super user name>).

11. Save the changes to the create_mysql_db.sh file and launch it to create the DB for Sailfish.

NOTE!: This procedure should be performed after each version update of Sailfish before launching the new version of it.

12. Launch the startup.sh file.

13. Sailfish GUI will be available via the following link in your browser: http://localhost:8080/demogui.

3.1.2 Windows

If you have the Windows operating system - you will need to run .bat batches instead of .sh ones. All other operations should be the same as in section 3.1.1.

3.2 Sailfish update

1. Shut down Sailfish by launching shutdown.bat/sh.

2. Create a back-up folder in Sailfish/.

3. Back up the files from <Deployed Tomcat>/webapps/demogui folder to the back-up folder (this allows using the current build if a new build has issues).

4. Download the sfgui.war archive installation file (Alternatively, you can request it from GitHub https://github.com/Exactpro/sailfish-core).

5. Extract the sfgui.war archive file to the <Deployed Tomcat>/webapps/demogui folder replacing the earlier files.

6. Start Sailfish by launching startup.bat/sh.

7. Sailfish GUI will be available via the following link in your browser: http://localhost:8080/demogui/.

3.3 Several Sailfish WEB UIs under one Tomcat

3.3.1 Linux

It’s required to configure a separate demogui folder in the Sailfish/apache-tomcat-8.5.33/webapps folder and configure separate DBs for each Sailfish.

1. To create a new DB, the create_mysql_db.sh file can be used (see section 3.1. point 8). Update the .sh file with the new DB name (e.g. sailfish2) and execute the .sh file:

2. Navigate to the Sailfish/apache-tomcat-8.5.33/webapps folder.

3. Copy the /demogui folder and rename it, e.g. demogui2.

4. Navigate to the demoqui2/cfg folder and update the hibernate.cfg.xml file as follows:

update the ‘hibernate.connection.url’ section with the corresponding new DB name:

5. Navigate to the Sailfish/apache-tomcat-x.x.xx/bin/ directory and launch the startup.sh file.

6. Both Sailfish UIs (demogui and demogui2) will be available in your browser:

demogui - on http://localhost:8080/demogui/

demogui2 - on http://localhost:8080/demogui2/

NOTE!: A restart of Tomcat triggers a restart of all Sailfish applications.

3.3.2 Windows

If you have the Windows operating system - you will need to run .bat batches instead of .sh ones. All other operations should be the same as in section 3.3.1.

4 Sailfish Web UI

4.1 Environment Page

The “Environment” tab in Sailfish, allows the user to manage the services and their parameters. “Services” in Sailfish are used for configuring settings for specific connections.

If there is a need to verify the same business scenario on another system – it is not necessary to re-build your test script – you can simply change the connection to the environment and run the existing script without any modifications.

A new service can be added by pressing the “Add” button:

Service parameters also need to be specified. There are Required and Optional parameters. The Required parameters will be taken by Sailfish as key values.

Below is the example of a Test Service Required and Optional parameters setup for a FIX client.

Once the environment is configured, the connection can be started by pressing “Start”.

If it is started successfully, you will see the “Started” status.

If the connection configuration is incorrect, the status will be ERROR.
If a connection can't be established at the current time, the status will be WARNING.

4.2 Test Script Page

The Test Scripts tab allows the user to perform the following:

1. Upload an existing matrix (a CSV/XLS/XLSX file): the user can see all uploaded matrices as a list; the name, date and time of uploading are indicated;

2. Create an empty form, so that the users’ own test cases could be written within Sailfish editors;
3. Manage the list of uploaded or created matrices, i.e. to delete, edit, and launch a few selected matrices, a single matrix, or a group of matrices for execution. There are two options for launching the matrices for executions in the management panel at the bottom of the page:

• • Continue if Failed (if disabled, Sailfish will stop executing a test script after the first failed case and skip all the remaining test cases. Otherwise, the test script will execute all test cases in a script irrespective of failures);
• • Autostart (if enabled, the test services required for execution of a test script will be started (and stopped after script execution) automatically. Otherwise, the user will have to start the services manually from the ‘Environment’ tab);

• If the user selects a single matrix, the following options are available:

• • Edit the matrix in Sailfish editors: as plain text/grid editor;
• • Edit the matrix in an Excel table;
• • Launch the matrix for execution.

• Either the entire matrix can be launched, or certain test cases within a matrix can be selected and launched for execution (in the text panel opposite the script name);

• View the list of matrices that have been run or enqueued for execution;
• For matrices that have been executed, you are able to view an executed test case report. The report contains step-by-step information about the outcome of each test case and each sent and received message.

4.3 Messages Page

The Messages tab allows the user to interact with the “Messages” table in the Sailfish database. The table is used for storing messages from all services. The working area of the page is divided into the following sections:

• The “Wizard” and “Query” fields are used to create HQL (Hibernate Query Language) queries (more information regarding HQL is available here: http://docs.jboss.org/hibernate/orm/3.3/reference/en/html/queryhql.html);
• The “Limit” field is used to set the user’s query limit;
• The “Columns” field (when clicked) provides a drop-down list with fields available for showing in UI;

• The ‘Show/Hide raw messages’ button to the left of each message opens a raw message in Hex and a human-readable format.

In order to create HQL queries using “Wizard”, the user should follow the steps below.

1. Click on the “Wizard” icon. The “Wizard” window opens:

2. Pick one of the two options available: “Add rule” and “Add Group”. “Add rule” allows adding expressions which are executed side by side. “Add Group” allows adding expressions which are executed in a pre-assigned order. Also, two operations can be used: “OR” and “AND”. To create a query, select the required options Add rule or/and Add Group in the upper right corner of the window and operations OR/AND in the upper left corner of the window (a single operation for all rules and a particular operation for every group).
3. Select a field in the table from the first drop-down list for every rule and group:

Where:

• Timestamp – the time when a message arrived;
• From/To – the service/gateway the message was sent to or received from;
• Admin – an administrative message;
• Namespace - used for technical needs;
• Name – the name of the message;
• HumanMessage – the contents of the message;
• RawMessage – a raw message in hex;

4. Select operators from the second drop-down list for every rule and group:

5. Enter a value (if required) into the last field for every rule and group;

6. Press the “Retrieve” button. The created query gets included into the “Query” field, and the messages in the table are sorted accordingly.

Example: It’s required to filter messages by Add Order (NOTE!: the message name should be written without spaces as a dictionary or script). The following rule can be used:

Then, it’s required to filter AddOrder messages from the User1 service. To achieve that, add a new rule with the following rule using operation AND:

After “Retrieve” is pressed, the messages are filtered by the following query: msg_name = 'AddOrder' AND from_service = 'User1'.

4.4 Download Page

All folders that exist on the local disk under <Deployed Tomcat>/webapps/sfgui/ are listed here and can be accessed from the Sailfish web GUI. The path to the folder can be specified in the “Path” field.

Once the folder is selected, the top panel gets activated.

You can navigate through folders by pressing to open the folder and pressing to go to the parent folder.

It is also possible to get the folder size by selecting the folder and pressing “Get folder size”. The size will appear on the right side of the selected folder in the “Size” column.

You can download the selected folder as .ZIP by pressing or download selected file by pressing . To show file content, press .

Note: Logs of Sailfish services can be found in ‘\logs\services<environment name> path’.

5 Test run results

Each Sailfish test script executed either automatically or manually will result in one of the following:

• Initialization failed (if Sailfish was not able to find a java source from a script due to wrong syntax);
• Run failed (if at least one service was not started by the “auto start” option);
• Passed (e.i. all verifications in the script matched the expected results);
• Failed (e.i. at least one verification in the test script did not match the expected result);
• Conditionally passed (e.i. at least one verification in the test script matched a result that was marked as a known issue in the system under test).

In each case (except script initialization failure, which will only provide an AML error), a detailed report will be generated automatically and stored in the default Sailfish ‘reports’ folder (the default location is <Deployed Tomcat>\webapps\sfgui\reports).

The report can be accessed either from the Sailfish UI ‘Test Scripts’ tab (the panel on the right) or from the “Script Run History” section of the Sailfish UI ‘Statistics’ tab.

The report can be downloaded as a .zip archive to the hard disk and browsed offline as a standard HTML file.

6 Test failures investigation

The Sailfish Regression Test library usually has the following hierarchy:

· Test Library consisting of

·· Test Scripts consisting of

···Test Cases consisting of

····Test Actions.

If any Test Action fails, its Test Case and Test Script fail too.

On the Test Script page, the report on the Sailfish ‘Test scripts’ tab is highlighted with:

• Red if the script failed due to INIT FAILED / RUN FAILED;
• Yellow if the script was EXECUTED but has some failed cases;
• Green if the script was EXECUTED, and all the test cases passed or conditionally passed.

The report can be downloaded as a .zip archive or opened on a new web page via the Report Link.

On the report page, there is a list of all test cases that are a part of the script, with their descriptions. They are highlighted with:

• Red for failed test cases;
• Blue for passed test cases;
• Yellow for conditionally passed test cases.

Left click on a row with a Test Case name. A detailed report will open with information as follows:

• Start Time;
• Finish Time;
• Unique Case ID (if filled in matrix);
• Hash of Unique Case ID;
• View menu which provides an ability to choose if all cases/fields should be displayed or only the fields;
• The list of Actions in the Test Case with their descriptions.

Left click on a row with a Test Action name. A detailed report will open with information as follows:

1. A table with Action details (list of Action Parameters);
2. Action Status with the failure reason;
3. A table with Similar messages (for Receive or Retrieve action);
4. Tables with parameters of Similar messages with Expected and Actual values for each parameter (Highlighted with red if there is a discrepancy).